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PREFACE 



This work was performed for the Under Secretary of Defense for Personnel and 
Readiness, Readiness and Training Directorate, under a task entitled “Development and 
Assessment of ADL Prototypes.” 

Despite the many common goals between the development of Advanced Distributed 
Learning (ADL) sharable instructional objects and intelligent tutoring systems (ITSs), this 
development is being pursued in parallel but independently. This document is intended to 
promote more joint effort between these two development communities and show why such 
joint development will benefit the Department of Defense (DoD) by reducing training and 
development costs and increasing the effectiveness of its training, education, and decision- 
aiding activities. 

An earlier version of this document was published in the International Journal of 
Cognitive Ergonomics and was reviewed by its editors and two anonymous reviewers. 
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EXECUTIVE SUMMARY 



The techniques of object-oriented programming are influencing many areas of soft- 
ware development, including the development of tools, materials, and applications for 
technology-based education and training. Reviews of technology-based instruction across 
many subject matters and settings have shown that it reduces costs by about a third and also 
either reduces the time to reach given instructional objectives by about a third or increases 
the achievement of students by about a third. An important benefit that technology-based 
instruction provides for military and industrial training is accessibility. It can be used 
effectively anytime and anywhere. 

These capabilities have motivated the Department of Defense (DoD), in cooperation 
with many other Federal agencies, to undertake an initiative in Advanced Distributed 
Learning (ADL). ADL combines the economic efficiencies of object-oriented instruction 
(which provides reusable, sharable instructional objects) and the learning efficiencies and 
accessibility of technology -based instruction. ADL is also targeting a third set of capabili- 
ties, namely those of intelligent tutoring systems (ITSs). 

ADL’s intention is to allow the real-time assembly, or generation, of instructional 
objects into instructional or decision-aiding presentations on demand (as needed by indi- 
vidual learners and users). It expects to capitalize on the unique characteristics of ITSs: 

• The ability to generate instruction, rather than require developers to foresee and 
pre-store all possible interactions between users and technology 

• The ability to support mixed initiative dialogue, in which either the user or the 
technology can initiate open-ended discussion. 

The ADL initiative and the development of ITSs, then, have several key goals in 
common: 

• Both are generative since they envision the generation of presentations on 
demand, in real time. 

• Both are intended to provide presentations that are tailored in content, sequence, 
level of difficulty, level of abstraction, style, and so forth to users’ intentions, 
backgrounds, and needs. 

• Both have a stake in research intended to accomplish such individualization. 
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• Both can be used equally well to aid learning or decision-making. 

• Both are intended to accommodate mixed initiative dialogue, in which either the 
technology or the user can initiate or respond to open-ended inquiries or dis- 
cussion. 

• Both will benefit greatly from a supply of sharable instructional objects readily 
available for the generation of instructional (or decision-aiding) presentations. 

Accordingly, ADL is focused on the specification of instructional objects that meet 
several criteria. Among these criteria, four seem most relevant: 

1. Accessible. It must be possible to find needed and sharable objects, and they 
must be accessible. Basically, we need widely accepted and standard ways to 
store objects so that widely accepted and standard ways can be used to find and 
retrieve them. 

2. Interoperable. Once found, the objects should be usable. This means that they 
must be interoperable and portable across most — if not all — ^platforms, 
operating systems, browsers, and courseware tools. 

3. Durable. Once implemented, the objects should continue to operate reliably. 
They should be durable. If the underlying platform, operating system, or 
browser is modified (e.g., when a new version is released and installed), 
courseware objects should continue to operate as before. 

4. Reusable. Finally, courseware objects should be reusable. Other platforms, 
operating systems, browsers, and courseware tools should be able to reuse, and 
perhaps even modify as needed, the original courseware objects. 

Specifications intended to meet these criteria are being developed through an 
evolving series of specifications called the Sharable Content Objects Reference Model 
(SCORM). Versions of SCORM can be found at the ADL web site ( www.adlnet.org ). 

We expect sharable instructional objects to reduce the costs and increase the 
effectiveness of ADL technologies and ITSs. Development of these objects should be 
cooperatively promoted and pursued by the ADL and ITS communities. Doing so will allow 
high-quality instruction and decision-aiding to become ubiquitous and affordable. 
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WHAT DO SHARABLE INSTRUCTIONAL OBJECTS 
HAVE TO DO WITH INTELLIGENT TUTORING SYSTEMS, AND 

VICE VERSA? 



The techniques of object-oriented programming are influencing many areas of soft- 
ware development. Among these areas are the development of tools, materials, and applica- 
tions for technology-based education and training. This document discusses the value of 
technology-based instruction in general, intelligent tutoring systems (ITSs) in particular, and 
how the specification and development of sharable instructional objects can enhance both. 

A. WHAT ARE THE CONTRIBUTIONS OF TECHNOLOGY-BASED 
INSTRUCTION? 

Substantial improvements in instructional effectiveness can be obtained by tailoring 
instruction to the needs and capabilities of individual learners. Studies of one-on-one 
tutoring provide evidence for these improvements. One widely cited discussion was based 
on studies performed by Benjamin Bloom. These studies compared the achievement of 
individually tutored students (one instructor for each student) with that of classroom stu- 
dents (one instructor for every 28-32 students) (Bloom, 1984). It is not surprising to find 
that individual tutoring in these studies increased the achievement of students. What is sur- 
prising is the magnitude of the increase. Bloom reported that the overall difference in 
achievement across three studies was about two standard deviations, which means, roughly, 
that tutoring improved the achievement of 50th percentile students to that of 98th percentile 
students. Two standard deviations is a large difference. Bloom posed it as a challenge to 
researchers and developers in education. 

Why is this 2-Sigma difference (as Bloom called it) a challenge? Why do we not 
simply provide one-on-one tutoring for all our students? The answer is straightforward and 
obvious: we cannot afford it. Providing one instructor for each student is, in most cases, 
prohibitively expensive. Because of this problem, Michael Scriven early on described indi- 
vidualized, tutorial instruction as both an instructional imperative and an economic impos- 
sibility (Scriven, 1975). 



We may now have the means to break out of this dilemma. Fundamental to the value 
of our rapidly evolving computer technology is its ability to adjust its actions in real time 
and on demand to the conditions of the moment. The promise of this capability for 
instruction has not been lost on researchers and developers. Tutorial instruction and tech- 
nology-based instruction use this ability to adjust actions responsively to meet the needs of 
individual students to solve two significant problems: individualized pace and interactivity. 
These problems confront all classroom teachers and may account for much of the difference 
in effectiveness between individualized, tutorial instruction and classroom instruction. 

1. The Challenges of Classroom Instruction: Pace 

The variability in time that different students take to leam and that teachers must 
accommodate in classrooms can be overwhelming. Despite conscientious efforts to sustain 
common levels of prior knowledge in classrooms, our current school practices appear to 
increase these differences by about one year for every year students spend in elementary 
school (Houston, 1997). For instance, the average spread of academic achievement in grade 
three is about 3 years. By grade six, it increases to about 6 years. 

Research has helped confirm the extent of this variability. Early on, Suppes (1964) 
found that the time needed by Kindergarten students to leam to build words from letters 
varied by about 13:1. Ratios for grade 5 students to master a unit of social studies were 
found to be 3:1 and 5:1 (Gettinger and White, 1980). In two different studies, the rates of 
learning observed among hearing-impaired and Native American students were found to be 
4:1 (Suppes, Fletcher, andZanotti, 1975 and 1976). Based on a range of research findings, 
Carroll (1970) estimated the overall ratio of 5:1 for elementary school students. Even among 
highly selected students at a major research university, the times needed by undergraduates 
to leam a programming language have varied by 7:1 (Corbett, 1998, personal commu- 
nication). The latter result helps confirm earlier observations: even though rates of learning 
may be determined initially by basic ability, the effects of ability are quickly overtaken by 
those of prior knowledge and experience with the subject matter (Tobias, 1989). 

What happens when we allow students to progress at their own rates, when we 
individualize the pace of learning as tutors do and as we have long been able to do using 
technology-based instmction? Some studies have compared the time to achieve specific 
instmctional objectives in classrooms with the time to achieve the same objectives using 
technology-based instmction. Reviews combining the results of these studies have been 
reported. Orlansky and String (1979) found that reductions in time to reach instmctional 
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objectives averaged about 54 percent in their review of computer-based instruction (CBI) 
used in military training. Fletcher (1997) reported an average time reduction of 31 percent in 
6 studies of interactive multimedia instruction applied in higher education. Kulik and his 
colleagues found time reductions of 34 percent in 17 studies of CBI used in higher educa- 
tion and 24 percent in 15 studies of adult education (Kulik, 1994). All these reviews are 
effectively independent since they reviewed different sets of evaluation studies. On this 
basis, reductions of about 30 percent in the time students take to reach a variety of given 
instructional objectives seem to be a reasonable expectation. 

The individualization of pace allowed by technology-based instruction, including 
ITSs, is likely to produce significant savings in time to learn. Do these savings in time 
matter? Studies have shown that the Department of Defense (DoD) might save hundreds of 
millions of dollars in specialized skill training if the training time to reach specified thresh- 
olds of knowledge and skill can be reduced by 30 percent (National Research Council, 
1997). These savings result as much from reductions in training costs as from the value of 
providing trained personnel ready for duty in operational units earlier. 

We might expect such savings to be substantial and operationally significant in 
military and industrial training, where the organization providing the training pays the stu- 
dents and where the employers and the students have a stake in the rapid mastery of subject 
matter and the early availability for organizational operations. However, these savings can 
also be equally valuable in K-12 education. As Gettinger and White (1980) asked, why 
should we assume that time to learn is unimportant to school children? All societies have a 
heavy stake in their success. Opportunities, such as those provided by technology-based 
instruction (i.e., permitting individual students to expand their capabilities and realize their 
potential as rapidly as possible), may be even more important than they are in military and 
industrial training. 

2. The Challenges of Classroom Instruction: Interactivity 

If we consider interactivity to be question-and-answer sequences occurring between 
students and instructors, could accounting for the differences we observe between one-on- 
many classroom instruction and one-on-one tutorial instruction also help? Those who study 
classroom interactions of this sort have found that groups of students ask about 3 questions 
an hour and that any single student in a class asks about 0.1 1 questions an hour (Graesser 
and Person, 1994). By contrast, students in individual tutorial sessions have been found to 
ask 20-30 questions an hour and have been required to answer 1 17-146 questions an hour. 
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Finally, some students taking CBI may answer 8-12 questions a minute. These questions 
have been selected to meet their individual needs and are immediately graded to provide 
instant feedback. 

The intensity of tutorial instruction provided by technology evidently pays off in 
student achievement. In a study that compared 97 empirical evaluations made up almost 
entirely of standard (non-ITS) CBI with conventional classroom approaches, Kulik (1994) 
reported an effect size advantage for computer-based approaches of about 0.39 standard 
deviations, or roughly an improvement of 50th percentile students to about the 65th per- 
centile of achievement. In an attempt to determine the advantages to instruction added by 
multimedia capabilities, Fletcher (1997) summarized a review of 47 comparisons of inter- 
active multimedia instruction with conventional classroom approaches and found an effect- 
size advantage for these technology-based approaches of about 0.50 standard deviations, or 
roughly an improvement of 50th percentile students to about the 69th percentile of 
achievement. Fletcher also reported on 1 1 ITS evaluations and found an effect size advan- 
tage for ITSs of about 0.84 standard deviations, roughly an improvement of 50th percentile 
students to about the 79th percentile of achievement. In a review of 5 evaluations of the 
SHERLOCK ITS system, Gott, Kane, and Lesgold (1995) report an overall effect size 
advantage of about 1.05 standard deviations, roughly an improvement of 50th percentile 
students to about the 85th percentile of achievement. 

We cannot say if we will achieve Bloom’s target improvement of two standard 
deviations, but the available evidence suggests that we are progressing in the right direction 
through the use of technology-based instruction. These findings also suggest that ITSs 
raise the ceiling for the effectiveness of technology-based instruction and provide new ave- 
nues for improvement over standard classroom instruction. We may be approaching the 
limits of the improvements we can achieve with general, non-ITS CBI; however, we also 
have much to learn about the instructional possibilities for ITSs. 

3. Are We Getting Anywhere?: Costs 

The preceding results show promise for technology-based instruction in general and 
ITSs in particular. However, if we want to make a difference in the day-to-day practice of 
instruction, we must address the concerns of those who are making decisions about it. For 
them, effectiveness is an important consideration but is only one component of the equation. 
The hallmark of administrative decision-making is the consideration of what must be 
surrendered to gain some advantage. Generally, these decisions involve tradeoffs between 
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costs and effectiveness. Decision-making in instruction is no different. We must consider 
costs and effectiveness. 

The ratios of costs for technology-based instruction compared with more conven- 
tional instruction can be reported in three categories: 

1. The initial investment costs to develop and implement both types of training 
(technology-based instruction and CBI) 

2. The operating and support costs for both types of training once they are in 
place 

3. These two cost categories combined. 

In reports structured for this purpose, the smaller the ratio, the better the news for 
the technology-based training. For one set of studies in which achievement under 
technology-based training was at least equal and generally superior to that produced by 
more conventional instruction, Fletcher (1997) reported the ratios to be 0.43 for initial 
investment, 0.16 for operating and support, and 0.35 overall. In these studies, substituting 
simulated equipment for the real equipment to be used on the job resulted in most of the 
cost savings achieved by technology-based instruction. Nonetheless, these ratios suggest 
that substantial economies can be realized through the use of technology-based instruction 
and that the return can justify the investment. More complete analyses, based on projections, 
suggest that the potential savings — not just cost avoidances — for DoD may amount to 
hundreds of millions of dollars (National Research Council, 1997). 

4. In Summary: The Thirds 

Assessments of technology-based instruction leave us with “the thirds.” Use of 
technology reduces the cost of instruction by about one-third, and it also either reduces time 
to reach given instructional objectives by one-third or it increases the achievement of its 
students by about one-third. The primary payoff for many organizations is, of course, the 
rapid preparation of personnel to perform operational duties. 

B. WHAT ARE THE CONTRIBUTIONS OF ITSs? 

It may be best to begin by noting the features that garden-variety CBI can and does 
provide. Notably it can: 

• Accommodate an individual student’s rate of progress, allowing as much or as 
little time that an individual student needs to reach instructional objectives 

• Adjust the sequence of instructional content to each student’s needs 
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• Adjust the content itself, ensuring that different students receive different con- 
tent depending on what they have mastered and what they have yet to learn 

• Make the instruction as easy or as difficult as necessary 

• Adjust to the learning style (e.g., verbal vs. visual) that is most appropriate for 
each student. 

These capabilities have been available and used in CBI since its inception in the 1950s (e.g., 
Atkinson and Fletcher, 1972; Suppes and Momingstar, 1972; Fletcher and Rockway, 1986). 
Those who promote systems with these features by touting them as indicators of newly 
developed “intelligent” capabilities may be missing some history. Whatever the case, they 
are using the term “intelligent” in ways that differ from the historical objectives of 
ITSs — objectives that have been pursued since the late 1960s. 

What are these objectives? What is left for ITSs to provide? What can we get from 
ITSs that is not otherwise available? Two functionalities deserve mention: 

1 . The ability to allow either the computer or the student to ask open-ended ques- 
tions and initiate instructional, “mixed-initiative” dialogue as needed or desired 

2. Related to the preceding functionality, the ability to generate instructional mate- 
rial and interactions on demand rather than require developers to foresee and 
pre-store all such materials and interactions needed to meet all possible eventu- 
alities. 

The first of these functionalities requires the ITS to understand and participate in 
mixed-initiative interactions with the student. It requires a mutual understanding of a lan- 
guage for information retrieval, decision-aiding, and instruction that is shared by the ITS 
and the student/user. Natural language has been a frequent choice for this capability, but the 
language of mathematics, mathematical logic, and electronics have been used, as reported in 
publications edited by Suppes (1981), Sleeman and Brown (1982), Psotka, Massey, and 
Mutter (1988), and Farr and Psotka (1992). 

Whatever form this dialogue takes, the mixed-initiative dialogue (in which either the 
student or the instructor can initiate interactions) appears to be a key feature of one-on-one 
tutorial instruction (e.g., Graesser, Person, and Magliano, 1995). In attempting to secure for 
our students the benefits of one-on-one tutorials, we may turn to technology-based 
instruction that specifically seeks to develop and implement mixed-initiative dialogue as an 
instructional approach. Such a capability has been a goal of intelligent tutoring systems 
(Carbonell, 1970) for a long time. 
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The second functionality requires ITSs to devise on demand — not retrieve from 
storage — interactions and presentations for individual students. This capability involves 
more than generating elements to fill in blanks in a template. It means generating interac- 
tions and presentations from information primitives using an “instructional grammar” that 
is analogous to the deep-structure grammar of the transformational-generative linguists of a 
generation ago. This functionality harkens back to the roots of ITS development, as (again) 
can be seen in the volumes edited by Suppes (1981), Sleeman and Brown (1982), Psotka, 
Massey, and Mutter (1988), and Farr and Psotka (1992). 

Motivations for both these functionalities can be found in basic research into human 
learning, memory, perception, and cognition. Findings from this research have led us to view 
all cognitive processes as constructive and regenerative. They have caused general theories 
of perception and learning to evolve from the fairly strict and logical positivism of 
behavioral psychology (which emphasized the study of directly observable and directly 
measurable actions) to a greater consideration of internal, less observable processes that are 
assumed to mediate and enable human learning and to produce the directly observable 
behavior that is the subject of behaviorist approaches. The keynote of these newer concep- 
tions of cognition may have been struck by Ulric Neisser who stated, “The central assertion 
is that seeing, hearing, and remembering are all acts of construction, which may make more 
or less use of stimulus information depending on circumstances.” (1967, p. 10). 

Neisser was led to this point of view by a large body of empirical evidence showing 
that many aspects of human behavior, such as seeing and hearing, simply could not be 
accounted for by external physical cues reaching human perceptors, such as eyes and ears. 
Additional processes, including an internally, one might say cognitively, generated analysis 
by synthesis, had to be posited to account for well-established and observable human abili- 
ties to detect, identify, and process physical stimuli. Such a process requires an active 
synthesis of the environment based on a runnable cognitive model, or simulation, that is 
validated or modified as needed by cues being received from sensory perceptors. It is the 
actively evolving simulation — not the stimuli alone — that is said to account for what the 
individual learns. 

These ideas were, of course, around long before Neisser published his book, and it 
is notable that they did not occur only to psychologists. Norbert Weiner, a cyberneticist, 
suggested that, “In both [the living individual and the machine], there exists a special appa- 
ratus for collecting information fi'om the outside world at low energy levels and for making 
it available for the operation of the individual or of the machine. In both cases, these external 
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messages are not taken neat, but through the internal transforming powers of the apparatus, 
whether it be alive or dead. The information is then turned into a new form available for the 
further stages of performance” (1954, pp. 26-27; italics are Neissers). 

All this leads to the impression that the generative capability sought by ITSs and by 
Advanced Distributed Learning (ADL) is not something merely nice to have but is essential 
if we are to advance beyond the constraints of the prescribed branching, programmed 
learning, and the ad-hoc principles currently used to design technology-based instruction. A 
generative approach is essential if we are to deal successfully with the immensity, extent, and 
variability of human cognition. 

The key defining characteristic of ITSs and ADL, then, is not the application of 
computer techniques from artificial intelligence (AI) or knowledge representation or the 
specification of sharable instructional objects, important as these may be. Rather, it is the 
functional capability to generate in real time and on demand instructional interactions that 
are tailored to student requests and/or needs. This generative capability motivated DoD to 
invest in ITSs originally (Fletcher and Rockway, 1986). At that time, the motivation was to 
reduce or eliminate the high costs of foreseeing or predicting all possibly needed materials 
and interactions, programming them, and pre-storing them into CBI. Today, this motivation 
remains fundamental to those who support ITSs and ADL development. 

1. An Example of an ITS at Work 

An example of these capabilities can be found in the student-computer, mixed- 
initiative dialogues supported by the Sophisticated Instructional Environment (SOPHIE), an 
early and seminal ITS (Brown, Burton, and DeKleer, 1982). SOPHIE used a general- 
purpose electronic simulator to represent an electronic power supply and create a reactive, 
tutorial environment for training in electronic troubleshooting. The underlying simulator 
allowed faults introduced into the system by a (human) instructor or by SOPHIE itself to be 
propagated throughout the (simulated) power supply in a realistic fashion. SOPHIE 
employed a semantic network of information about the power supply — one of the most 
sophisticated natural language understanders ever built into an ITS — and an articulate, 
expert troubleshooting capability that could not only solve arbitrary faults introduced by 
students or instructors, but also explain to the student how it solved them. 

SOPHIE’S reactive environment responded in a realistic fashion to the actions and 
decisions of a student engaged in practical problem-solving. It employed strategies that 
shadowed those used by the student to make inferences about the circuit based on the 
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student’s measurements. SOPHIE could thereby determine which deductions the student 
was using, construct a model of his/her state of knowledge, and use this model to generate 
explanations in terms that the student was prepared to understand. SOPHIE highlighted the 
need for intelligent training systems not only to solve arbitrary problems, but to do so in a 
way that can be explained to students and be understood by them. It pioneered capabilities 
currently lacking in many information acquisition and retrieval systems. 

The following is a student-computer dialogue adapted from SOPHIE: 

Student: What is the base emitter voltage of the voltage limiting 
transistor? 

SOPHIE: The base emitter voltage of Q5 is - 0.37 volts. 

The first student-SOPHIE interaction in this example is a straightforward question 
and answer. SOPHIE has introduced a fault into the simulated power supply and the stu- 
dent is trying to locate and repair it. 

Student: What about for the current limiting transistor? 

SOPHIE: The base emitter voltage of Q8 is .349 volts. 

The second student-SOPHIE interaction is a little less straightforward because, in 
addition to simulating the faulted power supply, SOPHIE must be able to parse the dialogue 
sufficiently to know that the student is asking for the base emitter voltage of transistor Q8. 
The meaning of the student’s question depends heavily on the context established by his 
prior question. 

Student: What should it be? 

SOPHIE: In a working circuit, the base emitter voltage of Q8 is .593 
volts. 

The third student-SOPHIE interaction continues its dependence on SOPHIE’S 
understanding of context and also requires SOPHIE to extract the answer by simulating 
both the faulted power supply and one that is operating correctly. 

Student: Replace Q8. 

SOPHIE: I am going to ask you some questions about how Q8 is 
faulted. Are any junctions shorted? 

The fourth student-SOPHIE interaction indicates a clear step beyond what Brown, 
Burton, and DeKleer described as a knowledgeable system to what they considered to be an 
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intelligent system. SOPHIE has progressed from a knowledgeable parsing of its dialogue 
with the student and simulation of various states of the power supply to a system exercising 
tutorial intelligence. It shadowed the student’s solution path, modeled the student’s 
troubleshooting hypotheses, determined that they are incorrect, elected to capture the 
dialogue initiative back from the student, and undertook a series of tutorial interactions 
intended to lead the student to a more correct approach to the problem. 

It is difficult to imagine any practical way to achieve this level of instructional func- 
tionality without the generative capability and mixed-initiative dialogue that distinguishes 
ITSs from other forms of CBI. 

2. What About Decision-Aiding? 

As can be seen, an ITS such as SOPHIE has many elements and capabilities in 
common with a decision-aiding system, intended in this case to assist an electronics mainte- 
nance technician. The differences may lie entirely in the intentions of the user and depend 
on whether the system is being used to solve a problem or to effect a (more or less) persis- 
tent change in the user’s knowledge and skills. The underlying computational capabilities 
appear to be the same. 

Is it worth expanding the capabilities of an ITS to support decision-aiding? The 
U.S. Air Force’s development and evaluation of the Integrated Maintenance Information 
System (IMIS) provided some evidence. Teitelbaum and Orlansky (1996) summarized this 
evidence. They reviewed and analyzed the performance of technicians who were specially 
trained to troubleshoot and maintain F-16 avionics systems and compared the performance 
of these technicians with the performance of less specifically (and much less expensively) 
trained General Aircraft Technicians. 

Table 1 shows the results of this comparison. Notice that the performance of the 
General Aircraft Technicians became essentially the same as that of the Avionics Specialists 
once they were trained on the IMIS. Perhaps more notable, however, is that the performance 
of the General Aircraft Technicians using IMIS was superior to that of Avionics Specialists 
using the paper-based Technical Orders currently provided as maintenance aids. Finally, it 
may be worth noting that because IMIS was tied into the inventory and parts databases, it 
could generate the paperwork needed to order new parts in about a minute for both groups. 
IMIS provided both sophisticated decision-aiding to the technicians as well as help in 
completing clerical tasks of a less exotic nature. 





Table 1. Results From IMIS Trials 
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IMIS supported decision-aiding only for three subsystems of F-I6 avionics. Even at 
that, if its use were expanded to the full fleet of F-16s for these three subsystems, net cost 
savings to the Air Force would be about $23 million annually. Therefore, using ITS 
techniques and capabilities to provide instruction and decision-aiding appears to offer sub- 
stantial benefits. 

The emphasis on IMIS as a decision-aiding system, rather than just a maintenance- 
aiding system, is intended to suggest that this approach is not limited to maintenance or 
operation of devices. Its capabilities could be applied to more abstract undertakings, such as 
command and control (C2) in the military and management and administration in the civilian 
sector. 

3. What is the Underlying Technical Idea? 

Achieving these instructional and decision-aiding functionalities is an important 
goal, but it requires a technical idea to make it feasible. Jaime Carbonell (1970) first articu- 
lated this idea. He recommended replacing the ad-hoc frame-oriented (AFO) instructional 
approaches, such as those seen in programmed texts and best described as “intrinsic pro- 
gramming” (Crowder, 1959), with an information-systems-orientated (ISO) approach. The 
AFO approach of intrinsic programming is widely used in technology-based instruction. It 
features a presentation (often a paragraph of text and/or a graphic of some sort) followed by 
questions. A student’s answers to these questions lead to different responses, usually dif- 
ferent branching paths, by the system. The presentations, the questions, and all possible 
branching paths must be anticipated and pre-specified by the instruction developers. 

Carbonell’ s ISO approach — or, as we might say today, an approach based on 
knowledge representation — allowed the instructional system to be generative. As suggested 
some time ago (Fletcher, 1975), the ISO functionalities possessed by SOPHIE and other 
ITSs involve computer representation or modeling of the student, the subject matter, and 
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expert tutoring. This point of view now seems to be commonly accepted and is found to a 
significant degree in the ITS architectures. Carbonell (1970) and Woolf and Regian (2000), 
among many others, have emphasized knowledge representation as a key underlying capa- 
bility for ITSs. Progress is being made. Woolf and Regian emphasize that our ability to 
represent human cognition has gained considerable potency because of the advances made 
in cognitive science during the last 10 years. These advances should substantially improve 
the promise and capabilities of ITSs. 

C. WHERE IS THIS TAKING US? 

1. The Third Revolution in Learning 

The emphasis here on instructional technology brings us to revolutions in instruc- 
tion. The first of these may have occurred with the development of written language about 
7,000 years ago. It allowed the content of advanced ideas and teaching to transcend time and 
place. The second revolution in instruction began with the technology of books. Books 
made the content of high-quality instruction again available anywhere and anytime but also 
inexpensive and accessible to many more people. A third revolution in instruction appears to 
be accompanying the introduction of computer technology. The capability of this tech- 
nology for real-time adjustment of instructional content, sequence, scope, difficulty, and 
style to meet the needs of individuals suggests a third pervasive and significant revolution in 
instruction. It makes both the content and the interactions of high-quality instruction widely 
and inexpensively accessible — again anytime, anywhere. 

2. The ADL Initiative 

Building on this possibility, the ADL initiative is now being undertaken in the 
United States and elsewhere. This initiative is intended to provide education, training, and 
decision-aiding (or “mentoring”) anytime, anywhere. It capitalizes on the growth of elec- 
tronic commerce and the World Wide Web. It takes advantage of this global — almost irre- 
sistible — activity, accelerates it, and applies it to learning. It will help ensure the availability 
of human competence in many organizations as they face the challenges of the 2T‘ century. 

The ADL initiative has been tasked to provide guidelines and specifications for all 
instructional activities of the United States Federal Government. It is similarly expected to 
provide guidelines and specifications for the instructional activities of many countries in 
Europe and in the Pacific Rim. It may succeed because it is not imposing standards devel- 
oped by the government, but, rather, is pulling together the best ideas, guidelines, and 
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specifications developed by a variety of industrial, academic, and government sources in the 
United States and elsewhere. It has become a massive cooperative development that involves 
economic sectors focused on achieving the ADL initiative’s goals, which are to provide 
learning and decision-aiding capabilities tailored to the needs of individuals anywhere and 
anytime they are needed. 

D. WHAT DOES ADL HAVE TO DO WITH ITSs? 

1. ADL and Intelligent Instructional Systems 

In the context of ITSs, “intelligent” refers as much to our intentions as our results; 
however, it is more than a marketing term. It refers to the specific functionalities that are the 
goals of ITS development. As discussed, these functionalities are distinct from those found 
in more conventional approaches to CBI. They require ITSs to generate instruction in real 
time and on demand as required by individual learners. This generative approach is also the 
goal of the ADL initiative, which is combining the benefits of both object-oriented 
development and Web delivery with those of technology-based instruction to achieve its 
objectives. 

Despite common goals, the ADL initiative and the development of intelligent 
instructional systems have been pursued in parallel but separately. The ADL initiative has 
focused on the specification of sharable instructional objects that can be retrieved from and 
then delivered over the World Wide Web anywhere, anytime. They can be used and reused 
to tailor instruction and decision-aiding presentations to the needs of individual learners. 
They support the core functional capability targeted by the ADL initiative: making the 
benefits of tutorial instruction and decision-aiding universally and affordably available. The 
common interests shared by the developers of ITSs and those pursuing the ADL initiative 
have been noted on both sides. 

The ADL initiative is guided by a vision, roughly illustrated in Figure 1, of a future 
in which everyone has an electronic personal learning associate. This device will assemble 
learning or decision-aiding presentations on demand and in real time — anytime, anywhere. 

The presentations will be exactly tailored to the needs, capabilities, intentions, and 
learning state of each individual or group (e.g., crew, team, or staff) of individuals. Com- 
munication with the personal learning associate will be based on natural language dialogue 
initiated either by the device or by its users. The device will be small enough to be carried in 
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Shareable courseware Assembled in To provide learning 

objects from across the real-time, on- and assistance 

World Wide Web demand anytime, anywhere 

Figure 1. An ADL Future 

a shirt pocket, or it will be wearable. It will be used by individuals learning by themselves, in 
groups, or in classrooms. It will, of course, be wireless. 

Most of the technology needed to build such a device exists now. Although we 
cannot yet fit it into a shirt pocket, advances in electronics should take care of that. What is 
especially needed for instruction and decision-aiding is content in the form of instmctional 
objects, which we are calling sharable instructional objects. These objects, shown in the 
cloud on the left side of Figure 1, must be readily accessible across the World Wide Web 
or whatever form our global information network takes in the future. 

Once these objects exist, they must be identified, selected, and assembled in real time 
and on demand and then handed to the personal learning associates, which provide the 
instmction or decision-aiding. This work of identifying, selecting, and assembling objects is 
the job of the server, represented as a box in the middle of Figure 1. By importing “logic” 
or instructional strategy objects, the server [which today would be called a Learning 
Management System (LMS)] can acquire the capabilities of the intelligent tutoring/ 
decision-aiding system we have been discussing. If it does so successfully, it will also 
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provide the benefits we have been discussing. However, in the ADL initiative, the underlying 
processes will rely on sharable instructional objects. 

The ADL initiative and the development of ITSs have several key goals in common: 

• Both are generative since they envision the generation of presentations on 
demand, in real time. 

• Both are intended to provide presentations tailored in content, sequence, level of 
difficulty, level of abstraction, style, and so forth to users’ intentions, back- 
grounds, and needs. 

• Both have a stake in the research intended to accomplish such individualization. 

• Both can be used equally well to aid learning or decision-making. 

• Both are intended to accommodate mixed initiative dialogue in which either the 
technology or the user can initiate or respond to inquiries in natural language. 

• Both will benefit greatly firom a supply of sharable instructional objects readily 
available for the generation of instructional (or decision-aiding) presentations. 

2. Sharable Instructional Objects 

Roschelle and Kaput (1995) emphasized the promise of object-based software, such 
as that promoted by ADL, for combining many kinds of interactive content in multiple 
display formats and attaining for education the benefits now being realized in business from 
the use of integrated office software. Roschelle et al. (1999) illustrated these points by 
examining the software techniques underlying five object-based education projects that have 
already been developed. Gibbons and Fairweather (2000) weigh these issues at length in 
examining the present status and future of CBI. 

The cloud on the left side of Figure 1 represents the World Wide Web or whatever 
we will use in the future to provide our global communication ether. One crucial matter for 
the implementation of ADL is what has loosely been called “content” in the form of 
sharable objects represented by the various icons shown in the cloud. People involved in 
ADL have spent — and continue to spend — much time, effort, and energy discussing these 
sharable content objects (SCOs). This matter transcends the immediate issues of ADL as 
evidenced by discussions edited by Wiley (2000) and others. 

As presently defined, SCOs could be entire courses, lessons within courses, or 
modules within lessons. They could be electronic representations of media, text, images, 
sound, Web pages, or other data that can be presented to students. The size, or 
“granularity,” of SCOs is a matter of considerable discussion. Gibbons, Nelson, and 
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Richards (2000) emphasized that SCOs will be most useful if they are prepared in 
sufficiently small chunks that can be accessed and reused by other instructional materials. 
Access must be rapid and easily accomplished across whatever form our future global 
information network takes. As suggested by Figure 1, once these SCOs exist, they must be 
available for assembly in real time and on demand by some sort of servers (middle of 
Figure 1) and then handed to client personal learning associates (right side of Figure 1). 

SCOs could also be material that is not seen by students but is needed to register 
them for courses, report on their progress, collect them into classes and other administrative 
groupings, or store data needed to tailor instruction to individual student needs. Sig- 
nificantly, they could also be content in the form of algorithms that aggregate, integrate, and 
sequence other objects as needed to manage the progress of students toward their attainment 
of specific instructional outcomes. 

Some researchers have provided examples and suggested some architectures for this 
possibility. Koedinger, Suthers, and Forbus (1999), in their discussion of tutor agents, 
emphasize their value for “higher-order” reasoning, such as that associated with developing 
and assessing experimental strategies, determining representation strategies, developing 
conjectures, drawing conclusions, and making arguments in a science learning space. Ritter 
and Koedinger (1996) suggest a general architecture that translates the mouse clicks, key 
presses, and other responses of students into semantic descriptions that can be used for 
tutoring and managing student progress toward curriculum goals. 

These activities are the kinds of functions that LMSs are now expected to perform. 
These LMSs are intended to be generalized capabilities for many different types of students, 
subject matters, curriculum structures, instructional approaches, and curriculum objectives. 
However, there seems to be a clear tradeoff between the amount of functionality that can be 
built into these LMSs and their generality. The more functionality built into them, the less 
general purpose they become. LMSs that import these functionalities as objects are likely to 
be more flexible, powerful, and generalizable. 

In this sense, the current discussion of “content” and what SCOs might be echoes 
the controversy that occurred early in the development of digital computers: whether data 
(traditional content) and logic (algorithms) should be stored in different ways and in 
different locations of the early machines (Goldstine, 1972). John von Neumann settled that 
discussion. He recommended storing data and logic together as digital bits in a common 
memory. The nature of SCOs, their disposition, and whether to apply a solution analogous 
to von Neumann’s remain to be determined. 
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Still, SCOs have become the foundation for ADL. The availability of SCOs will 
significantly reduce the costs of preparing instruction, decision-aiding, and job-performance 
assistance for technology-based delivery. This is likely to be true whether the SCOs are 
assembled in advance by course authors and developers or, as suggested by Figure 1, are 
assembled on demand and in real time by server algorithms incorporated in or imported into 
LMSs. For these reasons, ADL development is presently focused on packaging SCOs in 
anticipation of what is called by Spohrer, Sumner, and Shum (1998), among others, the 
“educational object economy.” The primary idea behind such an economy is that the 
emphasis in preparing materials for technology-based instruction (or decision-aiding) will 
shift from the current concern of preparing content components, or instructional objects, to 
one of integrating already available content into meaningful and relevant presentations. 

ADL developers recognize that this software engineering concern is only a begin- 
ning. The primary goal of ADL is not to promote tinkering with software objects, but to 
develop the functional capability of producing instructional outcomes anytime, anywhere 
they are needed or desired. The ADL initiative has made substantial progress at this soft- 
ware engineering level, but it must also address real learning issues. It must determine how 
to assemble instructional objects to achieve targeted instructional objectives. 

These two areas of software engineering and instructional design are not inde- 
pendent. They must coordinated and “harmonized” to achieve the ADL vision. This point 
suggests that designers and developers need some understanding of the underlying ADL 
architecture and its software specifications to ensure that ADL presentations bring about 
efficient and effective learning. It also suggests that the software engineers designing SCOs 
must (along with their meta-data packaging and assumptions concerning course structures, 
LMS structures, and communication protocols) understand and accommodate all varieties of 
instructional approaches and what each requires for software support. 

This vision, which foresees the development of an instructional object economy, 
must start with the specification of sharable instructional objects. Fortunately, many 
individuals (technicians, software engineers, instruction designers, cognitive researchers, and 
so forth) from organizations representing all sectors of the economy (government, industry, 
and academic) in the United States and elsewhere in Europe and Asia have joined in this 
quest. Development of these objects has become a global effort. Those involved in the U.S. 
ADL initiative mostly need to orchestrate the effort and document its results. 

Sharable instructional objects that support ADL functional requirements must meet 
some criteria. Among these criteria, four seem most prominent: 
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1. Accessible. It must be possible to find needed and sharable objects, and they 
must be accessible. Basically, we need widely accepted and standard ways to 
store objects so that widely accepted and standard ways can be used to find and 
retrieve them. 

2. Interoperable. Once found, the objects should be usable. This means that they 
must be interoperable and portable across most — if not all — ^platforms, 
operating systems, browsers, and courseware tools. 

3. Durable. The objects should be durable. Once implemented, the objects 
should continue to operate reliably. If the underlying platform, operating 
system, or browser is modified (e.g., when a new version is released and 
installed), courseware objects should continue to operate as before. 

4. Reusable. Finally, courseware objects should be reusable. Other platforms, 
operating systems, browsers, and courseware tools should be able to reuse, and 
perhaps even modify as needed, the original courseware objects. 

Specifications intended to meet these criteria are being developed through an 
evolving series of specifications called the Sharable Content Objects Reference Model 
(SCORM). Versions of SCORM can be found at the ADL website ( www.adlnet.org ). 

Specifying these objects arises fi'om the intersection of disciplines concerned with 
learning and cognition, information storage and retrieval, and software engineering. 
Packaging the objects so that they are accessible will depend on the model of cognition and 
learning or the learning strategy trying to find them. This packaging requires the develop- 
ment of a taxonomy that is sufficiently robust to accommodate many models and strategies. 
Aside from such models of human cognition, learning, and instruction, some representation 
of the physical world is also required to accommodate comprehensive range of objects. 
Finally, interoperability and reusability require techniques of software engineering that are 
compatible with the prevailing global ether — which today is the World Wide Web — and 
with the many operating environments in which the instructional objects will be used. Given 
the difficulty of the undertaking, it is fortunate that it has become a global quest. Current 
progress toward these ends can be viewed through many portals. The ADL website is a 
good a place to start. 

E. SUMMARY 

This discussion suggests that: 

• Substantial improvements in instructional effectiveness can be obtained by 
tailoring instruction to the needs and capabilities of individual learners. Evi- 
dence for these improvements can be found in studies of one-on-one tutoring. 
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However, in most applications, providing one instructor for each student is 
prohibitively expensive. 

• Using technology can make the benefits of individualized instruction afford- 
able. Evaluations of CBI in general and ITSs in particular provide evidence of 
these benefits. 

• Combined with the availability of sharable instructional objects available from 
the Web, ITSs will lead to the development and wide use of personal learning 
associates, which will allow high-quality instruction and decision-aiding to 
become ubiquitous and affordable. 

• This is a desirable result. 

If Kurzweil’s (1999) projections are correct, computers may become more effective 
in providing instruction than human tutors even if humans use all the techniques Graesser 
and his colleagues found they now neglect (Graesser, Person, and Magliano, 1995). We 
may not be implanting integrated circuits in our brains as Kurzweil suggests, but the goal of 
using technology to discover more than any human agent can about the unique potential of 
every individual and devising effective and individualized procedures to reach it seems an 
appealing and realistic prospect. 
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